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Abstract Body 


Background / Context: 

A consistent finding across early childhood interventions targeting mathematics and other 
achievement-related skills that is that initial treatment effects fade over time, with children not 
receiving the intervention catching up to children who did (Clements et ah, 2013; Leak et ah, 
2010; Puma et ah, 2010). One popular explanation for this is that, owing to their own preparation 
or a desire to ensure that all children receive basic skills training, elementary school teachers do 
not teach the kind of advanced content that children are prepared for after receiving an effective 
early mathematics intervention (Bodovski & Farkas, 2007; Crosnoe et ah, 2010; Engel, 
Claessens, & Finch, 2014). Engel and colleagues (2014) found that most kindergarten teachers 
taught mathematics content at the level appropriate only for the lowest achieving students. 
Teachers reported spending more time on basic skills, such as counting and shape recognition, 
than any other skills, even though the majority of the children had already mastered these skills 
at school entry. A lack of exposure to advanced mathematical content may impose a ceiling on 
higher achieving children’s subsequent mathematics achievement trajectories. Thus, children 
who benefit from early mathematics interventions may experience flatter achievement 
trajectories in subsequent years. In contrast, the lower achieving children who did not receive the 
intervention may benefit more from the basic level of instruction and catch up to their higher 
achieving peers. We refer to this hypothesis for fadeout of treatment effects in early mathematics 
interventions as the “same old schools” hypothesis. 

An alternative hypothesis is short-run differences in mathematics achievement induced 
by interventions are soon overtaken by differences produced by relatively stable factors 
influencing children’s long-tenn mathematical development. These factors likely include a 
combination of domain general cognitive abilities, relatively stable academic skills, motivation, 
and other factors that affect children’s mathematics achievement across time. Consistent with 
this hypothesis, Bailey and colleagues (2014) found that correlations between children’s 
mathematics achievement scores at different times during their mathematical development were 
accounted for in greater part by relatively stable factors across development than by unique 
effects of previous achievement on later achievement, though both types of effects were 
observed. If this is true, then children’s higher mathematics achievement at the end of an 
effective early mathematics intervention might subsequently migrate to levels that would be 
predicted based on their underlying skills alone. We refer to this hypothesis for fadeout of 
treatment effects in early mathematics interventions as the “underlying skills” hypothesis. 

Purpose / Objective / Research Question / Focus of Study: 

The same old schools and underlying skills hypotheses generate different predictions 
about children’s subsequent achievement trajectories following the conclusion of an effective 
early mathematics achievement intervention. The same old schools hypothesis posits that a lack 
of advanced instruction in early schooling imposes a ceiling on higher achieving children’s 
mathematics learning. This ceiling should be present both for children whose skills were boosted 
by the intervention as well as control-group children who perform similarly to treatment-group 
children following the intervention. 

The underlying skills hypothesis presumes that children in the control group with similar 
scores on the post-intervention assessment to children in the treatment group are advantaged on 
underlying skills, having reached the same level of achievement without the benefit of the 
effective early mathematics intervention. If this is true, the relatively high-achieving control 
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students who enter kindergarten with similar skills to their peers in the treatment group are 
predicted to experience greater achievement gains during the kindergarten year. We illustrate the 
conflicting predictions of the same old schools hypothesis and the underlying skills hypothesis in 
Figure 1. 

(insert Figure 1 here) 

We tested these two hypotheses using data from an effective early mathematics 
intervention. Specifically, we identified a subset of children in the control group with similar 
post-intervention achievement scores to children in the treatment group, and then tested whether 
their subsequent learning trajectories differed. 

Setting: 

The current study uses data from the Technology-enhanced, Research-based, Instruction, 
Assessment, and professional Development (TRIAD) intervention evaluation (Clements et ah, 
2011; Clements, Sarama, Wolfe, & Spitler, 2013), a study implemented in 42 low-resource 
schools designed to investigate the effects of the Building Blocks early mathematics curriculum. 
The intervention is described in greater detail below. 

Population / Participants / Subjects: 

At the intervention’s conclusion, children were assessed immediately thereafter and then 
one year later in the kindergarten year. The overall sample includes 361 children in the control 
group and 418 children in the treatment group who had valid mathematics achievement score 
data in pre -kindergarten (the post- test) and kindergarten (the follow-up assessment). Children 
came predominantly from low income families: Eighty-five percent of students qualified for free 
or reduced price lunch. Descriptive statistics for this sample are presented in Table 1. 

(insert Table 1 here) 

Intervention / Program / Practice: 

The intervention in the pre -kindergarten year included the Building Blocks mathematics 
curriculum and professional development for teachers in schools assigned to the treatment 
condition 1 . The control condition received business as usual. The curriculum produced 
substantial gains in the treatment group, and the treatment effect persisted into the third year of 
the study, but the effect diminished over time (Clements et ah, 2013). Additional details about 
the Building Blocks curriculum are available in Clements & Sarama (2007). 

Research Design, Data Collection, and Analysis: 

Children’s mathematics achievement was assessed immediately following the 
intervention (post-test) and then one year later in the kindergarten year (follow-up assessment), 
using the Research-based Early Mathematics Assessment (REMA). The measure is well 
validated and specifically designed for early mathematics (Clements, Sarama, & Liu, 2008). This 
measure defines mathematics achievement as a latent trait using the Rasch model, producing a 
score with a consistent metric that places children on a common ability scale (Bond & Fox, 

2001; Clements et ah, 2011). 


In the original TRIAD study, schools were randomly assigned to three conditions: (1) pre-kindergarten only 
treatment; (2) pre-kindergarten to first grade treatment; (3) control condition. Our analysis sample only included 
children who were in the pre-kindergarten only treatment and the control condition, since the largest fadeout effect 
occurred between these two groups. 
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To test both the underlying skills and same old schools hypotheses, we matched the 
treatment group to a subset of the control group with a similar mean and variance of mathematics 
achievement scores at the post-test. We then compared these groups’ mathematics achievement 
scores at the follow-up assessment. The same old schools hypothesis predicts that these groups 
will score similarly at the follow-up assessment, while the underlying skills hypothesis predicts 
that children in the matched control group will outperform children in the matched treatment 
group. 

To further test the underlying skills hypothesis, we then tested whether differences 
between the matched groups at the follow-up assessment were attenuated by a series of 
predictors. We regressed follow-up mathematics achievement on matched group and the 
following sets of predictors, adjusting for child age at the initial assessment and weighting by 
likelihood of being in the treatment group. 

We tested whether differences between the matched groups at the follow-up assessment 
were attenuated by measures of socioeconomic status. Previous research suggests that poverty 
influences to children’s academic achievement (Duncan et ah, 1998). Therefore, it is possible 
that children in the matched control group outperform children in the matched treatment group 
on the follow-up assessment, because the matched control children come from less 
disadvantaged homes. Child covariates included whether the child received free or reduced price 
lunch measured in the kindergarten year (dummy coded as 0 or 1) and mother’s highest level of 
education. For mother’s level of education, we created dummy coded variables indicating the 
highest level of formal schooling the mother attained: no high school, high school, some college, 
college, or missing (with the mother attaining a high school diploma as the reference group). 

To test the underlying skills hypotheses, we tested whether differences between the 
matched groups at the follow-up assessment were attenuated by children’s literacy and language 
skills. Group differences in children’s language and literacy skills might statistically contribute 
to higher achievement in matched controls on the follow-up assessment, because these skills may 
directly affect children’s later achievement trajectories, and are likely correlated with domain 
general cognitive abilities (Deary, Strand, Smith, & Fernandes, 2007). Literacy skills were 
measured in the spring of kindergarten with the PALS - PreK (Invernizzi, Sullivan, Swank, & 
Meier, 2004) and MCLASS: CIRCLE (Landry, 2007). Both are commonly used literacy 
measures for early childhood programs, measuring recognition of upper and lowercase letters. 
Language skills were measured in the fall of kindergarten with the Renfrew Bus Story - North 
American Edition (Glasgow & Cowley, 1994). 

Finally, we tested whether preschool fixed effects attenuated differences between the 
matched groups at the follow-up assessment. Before the study began, preschools in these two 
groups were grouped into blocks containing two of these schools with similar scores, and then 
schools were randomly assigned to the treatment or control group within block. Therefore, it is 
possible that children in the matched control group outperform children in the matched treatment 
group on the follow-up assessment, because their higher quality prior school experiences have 
better prepared them for future learning. 

Findings / Results: 

In the full sample, children in the treatment group outperformed children in the control 
group at the post-test assessment ( p < .001). This effect faded out but remained statistically 
significant at the follow-up assessment (p < .05; Figure 2). 

(insert Figure 2 here) 
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In the matched sample, children in both groups performed identically at the post-test 
assessment, and the matched control group significantly outperformed the matched treatment 
group at the follow-up assessment (p < . 001; Figure 3). This pattern of results is consistent with 
the underlying skills model (Figure 1). 

(insert Figure 3 here) 

Table 2 shows the results of the regression models testing for which variables attenuate 
the difference between the matched groups at the follow-up assessment. The matched groups 
differed, adjusting for age (Model 0). The magnitude of this difference was reduced by 27% after 
adjusting for socioeconomic status (Model 1). Adjusting for language and literacy reduced the 
group difference by 95%, at which point it was no longer statistically significant (Model 2). 
Adjusting for preschool block reduced the group difference by 27%. When socioeconomic status, 
language, literacy, and preschool block were all included in the same model, adjusting for age, 
only literacy and language scores had statistically significant effects on children’s follow-up 
mathematics achievement, and the difference between the matched groups was completely 
eliminated (Model 4). Taken together, these results suggest that children’s language and literacy 
scores accounted for the difference between the matched groups’ mathematics achievement at 
the follow-up assessment. 

(insert Table 2 here) 


Conclusions: 

Children in the control group with similar post-test scores to children in the treatment 
group subsequently outperfonned children in the treatment group, and this difference was almost 
completely accounted for by language and literacy, measures of children’s underlying skills. 
Results support the underlying skills hypothesis account of fadeout of treatment effects for 
effective early mathematics interventions. 

Though the same old schools hypothesis and the underlying skills hypothesis differ in 
their accounts of the processes underlying fadeout of treatment effects in early mathematics 
interventions, they have somewhat similar implications for boosting low achieving children’s 
long-term achievement outcomes. Both hypotheses suggest the importance of changes in 
children’s learning environments for sustaining treatment effects at their initial levels. The key 
difference between the two hypotheses relates to the type of instructional change likely to be 
most effective. The same old schools hypothesis suggests that treatment effects will be sustained 
if children are exposed to more advanced content in school after the intervention concludes. In 
contrast, the underlying skills hypothesis suggests two possible solutions. First, this hypothesis 
suggests that identifying intervening on non-mathematical skills that have relatively stable 
effects on children’s mathematics achievement throughout development may reduce fadeout. 
Also, the underlying skills hypothesis suggests that an effective treatment might include 
additional high quality intervention for low achieving children, even after the conclusion of the 
initial effective early mathematics intervention. In other words, the most effective way to 
maintain relatively high levels of mathematics achievement produced by high quality 
interventions, such as Building Blocks, may be subsequent high quality interventions. 
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Appendix B. Tables and Figures 


Table 1 


Sample characteristics before and after matching on post-test mathematics 
achievement 



Treatment Mean (SD) or 

Control Mean (SD) or 


Proportion 

Proportion 

Variable 

Before After 

Matching Matching 

Before After 

Matching Matching 


Mathematics achievement 


Pre-test 

-3.25 (.86) 

-3.45 (.86) 

-3.17 (.80) 

-3.00 (.74) 

Post-test 

-1.86 (.65) 

-2.08 (.59) 

-2.27 (.75) 

-2.08 (.62) 

Follow-up 

-1.07 (.66) 

-1.19 (.63) 

-1.19 (.69) 

-1.07 (.65) 

Child characteristics 

Proportion Male 

.50 

.53 

.50 

.49 

Age (in years at first pre-K assessment) 

4.33 (.35) 

4.29 (.36) 

4.39 (.35) 

4.40 (.35) 

Proportion free and reduced price lunch 

.82 

.86 

.88 

.86 

Language score (kindergarten) 

-.78 (.87) 

-.92 (.82) 

-.99 (.93) 

-.84 (.88) 

Literacy score (kindergarten) 

.55 (.28) 

.51 (.28) 

.56 (.26) 

.60 (.25) 

Maternal education (Proportions) 

Mother's education: No high school 

.14 

.16 

.15 

.13 

Mother's education: High school 

.32 

.31 

.31 

.30 

Mother's education: Some college 

.39 

.44 

.33 

.36 

Mother's education: College 

.15 

.09 

.20 

.20 
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Table 2 

Models testing for attenuation of the effect of matched group on follow-up mathematics 
achievement score 



Model 0 

Model 1 

Model 2 

Model 3 

Model 4 

Matched group 

-.22** 

-.16* 

-.01 

-.18* 

.02 


(.07) 

(.07) 

(.08) 

(.08) 

(.08) 

Age at first test 

25*** 

25*** 

19*** 

.26*** 

lg*** 


(.04) 

(.03) 

(.04) 

(.04) 

(.04) 

Free or reduced price lunch 


.04 



-.12 



(.12) 



(.12) 

Mother's education: no high 
school 


-.17 



-.07 



(.13) 



(.14) 

Mother's education: some 
college 


-.15 



-.13 



(.10) 



(.10) 

Mother's education: college 


41** 



.13 



(.13) 



(.14) 

Kindergarten literacy score 



35*** 


35*** 




(.04) 


(.04) 

Kindergarten language score 



2i*** 


22*** 




(.04) 


(.04) 

Block fixed effects 




Inc. 

Inc. 

Missing dummy variables 


Inc. 



Inc. 

R 2 

.10 

.18 

.32 

.17 

.37 

Observations 

576 

576 

397 

576 

397 


Note. Standard errors in parentheses. All models are weighted likelihood of being in the 
treatment group. Missing dummy variables were included in the analyses to account for 
missing data on the socioeconomic status measures. Mathematics achievement 
outcome, literacy, and language scores were standardized to have a mean of 0 and 
standard deviation of 1 . 

*p<.05, **p<.01; *** p<.001 
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Figure 1 

Predicted Trajectories for Treatment and Higher Achieving Controls based on the Same Old 
Schools and Underlying Skills Hypotheses 


Same Old Schools Underlying Traits 




Note. The gray line in the left panel represents a ceiling imposed by the low-level content of 
instruction, which constrains the later trajectories of children in the treatment group, according to 
the same old schools hypothesis. The gray line in the right panel represents the trajectory the 
treatment group would have been predicted to follow, had they not received treatment. 
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Figure 2 

Mathematics Achievement Trajectories of the Intervention Groups Before Matching on Post-Test 
Mathematics Achievement 


Before matching 



Figure 3 

Mathematics Achievement Trajectories of the Intervention Groups After Matching on Post-Test 
Mathematics Achievement 


After matching - without replacement 



Time 
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